Interesting approach to parallelism, what would be a concrete use case for only letting information flow one way from the sidekick to the http workers?

Usually the flow would be inverted, where a http worker offloads work to a pool of 'sidekick' workers and can optionally wait for a task to complete.

Thank you for the contribution. Interesting idea, but I'm thinking we should merge the approach with #1883. The kind of worker is the same, how they are started is but a detail.

@nicolas-grekas the Caddyfile setting should likely be per php_server, not a global setting.

@AlliBalliBaba The use case isn't task offloading (HTTP->worker), but out-of-band reconfigurability (environment->worker->HTTP). Sidekicks observe external systems (Redis Sentinel failover, secret rotation, feature flag changes, etc.) and publish updated configuration that HTTP workers pick up on their next request; with per-request consistency guaranteed via $_SERVER injection. No polling, no TTLs, no redeployment.

Task offloading (what you describe) is a valid and complementary pattern, but it solves a different problem. The non-HTTP worker foundation here could support both.

@henderkes Agreed that the underlying non-HTTP worker type overlaps with #1883. The foundation (skip HTTP startup/shutdown, immediate readiness, cooperative shutdown) is the same. The difference is the API layer and the DX goals:

• Minimal FrankenPHP config: a single sidekick_entrypoint in php_server(thanks for the idea). No need to declare individual workers in the Caddyfile. The PHP app controls which sidekicks to start via frankenphp_sidekick_start(), keeping the infrastructure config simple.

• Graceful degradability: apps should work correctly with or without FrankenPHP. The same codebase should work on FrankenPHP (with real-time reconfiguration) and on traditional setups (with static or always refreshed config).

• Nice framework integration: the sidekick_entrypoint pointing to e.g. bin/console means sidekicks are regular framework commands, making them easy to develop.

Happy to follow up with your proposals now that this is hopefully clarified.
I'm going to continue on my own a bit also :)

Great PR!

Couldn't we create a single API that covers both use case?

We try to keep the number of public symbols and config option as small as possible!

@henderkes Agreed that the underlying non-HTTP worker type overlaps with #1883. The foundation (skip HTTP startup/shutdown, immediate readiness, cooperative shutdown) is the same. The difference is the API layer and the DX goals:

Yes, that's why I'd like to unify the two API's and background implementations into one. Unfortunately the first task worker attempt didn't make it into main, but perhaps @AlliBalliBaba can use his experience with the previous PR to influence this one. I'd be more in favour of a general API, than a specific sidecar one.

The PHP-side API has been significantly reworked since the initial iteration: I replaced $_SERVER injection with explicit get_vars/set_vars protocol.

The old design used frankenphp_set_server_var() to inject values into $_SERVER implicitly. The new design uses an explicit request/response model:

• frankenphp_sidekick_set_vars(array $vars): called from the sidekick to publish a complete snapshot atomically
• frankenphp_sidekick_get_vars(string|array $name, float $timeout = 30.0): array: called from HTTP workers to read the latest vars

Key improvements:

• No race condition on startup: get_vars blocks until the sidekick has called set_vars. The old design had a race where HTTP requests could arrive before the sidekick had published its values.
• Strict context enforcement: set_vars and should_stop throw RuntimeException if called from a non-sidekick context.
• Atomic snapshots: set_vars replaces all vars at once. No partial state possible
• Parallel start: get_vars(['redis-watcher', 'feature-flags']) starts all sidekicks concurrently, waits for all, returns vars keyed by name.
• Works in both worker and non-worker mode: get_vars works from any PHP script served by php_server, not just from frankenphp_handle_request() workers.

Other changes:

• sidekick_entrypoint moved from global frankenphp block to per-php_server (as @henderkes suggested)
• Removed the $argv parameter: the sidekick name is the command, passed as $_SERVER['argv'][1]
• set_vars is restricted to sidekick context only (throws if called from HTTP workers)
• get_vars accepts string|array: when given an array, all sidekicks start in parallel
• Atomic snapshots: set_vars replaces all vars at once, no partial state
• Binary-safe values (null bytes, UTF-8)

Thanks @dunglas and @henderkes for the feedback. I share the goal of keeping the API surface minimal.

Thinking about it more, the current API is actually quite small and already general:

• 1 Caddyfile setting: sidekick_entrypoint (per php_server)
• 3 PHP functions: get_vars, set_vars, should_stop

The name "sidekick" works as a generic concept: a helper running alongside. The current set_vars/get_vars protocol covers the config-publishing use case. For task offloading (HTTP->worker) later, the same sidekick infrastructure could support:

• frankenphp_sidekick_send_task(string $name, mixed $payload): mixed
• frankenphp_sidekick_receive_task(): mixed

Same worker type, same sidekick_entrypoint, same should_stop(). Just a different communication pattern added on top. No new config, no new worker type.

So the path would be:

1. This PR: sidekicks with set_vars/get_vars (config publishing)
2. Future PR: add send_task/receive_task (task offloading), reusing the same non-HTTP worker foundation

The foundation (non-HTTP threads, cooperative shutdown, crash recovery, per-php_server scoping) is shared. Only the communication primitives differ.

WDYT?

I think the failures are unrelated - a cache reset would be needed. Any help on this topic?

Hmm, it seems they are on some versions, for example here: https://github.com/php/frankenphp/actions/runs/23192689128/job/67392820942?pr=2287#step:10:3614

For the cache, I'm not aware of a Github feature that allow to clear everything unfortunately 🙁

The name "sidekick" works as a generic concept: a helper running alongside. The current set_vars/get_vars protocol covers the config-publishing use case. For task offloading (HTTP->worker) later, the same sidekick infrastructure could support:

My only worry with this is that "sidekick" implies that there's a "main" character related to it. That's the case here, but wouldn't necessarily be the case for task- or extension workers.
Think of something like
task_worker bin/console messenger:consume scheduler_default async -vv

Other than the naming, I don't object the api.

If we want to unify these concepts, the sidekick workers should probably be configured like regular workers and started always from the Caddy config (very messy to let http workers start and stop them). Just a worker instead of sidekick_entrypoint and mark the worker as non_http somehow.

The frankenphp_sidekick_set_vars and frankenphp_sidekick_get_vars are essentially just a process-wide store, you could do the same with something like apcu. I think having global variables for ZTS could be very useful, but it should be well thought through since there many potential use cases like caches or locks. You probably also need ways to flush and debug the store, like apcu offers. Might make sense to have something like a FrankenPHP\Store or so and support more than just strings.

If we want to unify these concepts, the sidekick workers should probably be configured like regular workers and started always from the Caddy config (very messy to let http workers start and stop them). Just a worker instead of sidekick_entrypoint and mark the worker as non_http somehow.

Absolute agree, though I believe it would be good to be able to start them from http code and establish communication through channels of sort.

The frankenphp_sidekick_set_vars and frankenphp_sidekick_get_vars are essentially just a process-wide store, you could do the same with something like apcu. I think having global variables for ZTS could be very useful, but it should be well thought through since there many potential use cases like caches or locks. You probably also need ways to flush and debug the store, like apcu offers. Might make sense to have something like a FrankenPHP\Store or so and support more than just strings.

Funny you say that, I created one over the last few days, but I quickly figured that the Cgo overhead and Go generally being much slower than optimised C made this a bit of a futile attempt. Ristretto backend was ~4x slower than apcu/direct C copying. If we want to implement a FrankenPHP\Store it will have to be well-thought out and implemented in pure C.

All green, ready on my side!🎳

@henderkes Thanks for validating the CGo overhead concern. A proper FrankenPHP\Store would need pure C and is a separate, much larger project. The sidekick API is intentionally narrow: lifecycle management + config publishing.

Glad we agree on starting from PHP! That's the current design: get_vars implicitly starts the sidekick on first call.

@AlliBalliBaba About APCu: I want sidekicks to be a core feature of FrankenPHP that people can reliably build on, not something depending on an optional third-party extension with its own bugs and serialization overhead. The sidekick API solves a different problem than a shared store: lifecycle management (start, wait-for-ready, crash recovery, shutdown), blocking first call (no polling), and per-sidekick scoping. String-only is by design; config updates don't need complex types, and explicit serialization is always available.

"very messy to let http workers start and stop them"

It's actually clean. Every HTTP worker has a bootstrap phase before entering the frankenphp_handle_request loop. That's where get_vars should be called. At-most-once semantics make it safe: all workers call get_vars('redis-watcher') during boot, only the first starts the sidekick, all block until it's ready. Sidekicks are never stopped from PHP; they live for the process lifetime and shut down cooperatively via should_stop().

About naming: my preference goes to "sidekick" precisely because these workers ARE secondary to the main app. A Messenger consumer is a different pattern (standalone queue processor). Sidekicks observe the environment and support the main app. The name is also memorable and distinct from the existing worker concept.

Caddy-config-based workers could be added later as a complementary approach for ops-level control, although I'm not sure that'd provide the best DX for app developers. Those will prefer not touching the Caddyfile when adding a new sidekick (eg when installing a third-party-provided one).

About APCu: I want sidekicks to be a core feature of FrankenPHP that people can reliably build on, not something depending on an optional third-party extension with its own bugs and serialization overhead

Not saying it must be apcu and it would also be fine to start off with string-only support. Just saying that you can't change the API afterwards without BC break, so the functions/classes added to core should be well-named and well thought out so there's room for extension and optinization.

"very messy to let http workers start and stop them"

It is messy since the http-workers can start these background threads at runtime without any mechanism to oversee or stop them. It can become hard to reason about which sidekick workers are currently running.
If you define them via config, you can be sure: "there is exactly one redis discovery worker active in this process at all times" by just looking at the config. You can even have a pool of worker threads and different entry-scripts.

The original concept of 'task workers' had workers call frankenphp_handle_task() or frankenphp_handle_request() to mark them as ready (requests are not received until all workers are ready). Other threads then could send a task to the worker. In your case, the worker could just be pinged periodically from the go side with a task to discover the redis endpoint.

The original concept of 'task workers' had workers call frankenphp_handle_task() or frankenphp_handle_request() to mark them as ready (requests are not received until all workers are ready). Other threads then could send a task to the worker. In your case, the worker could just be pinged periodically from the go side with a task to discover the redis endpoint.

Yes, I specifically like this approach more because it can handle many different types of workers (queue consumers, publishers, sidekicks and extension workers) with a unified C/Go API.

It is messy since the http-workers can start these background threads at runtime without any mechanism to oversee or stop them.

I believe the point is to define them as always available in a scripts lifetime after being started. A declaration in the Caddyfile would necessitate it to live the entire time and start immediately.

I do think we should allow them to communicate with all serializables, though, not just strings, but could be added later on.

Thanks for the review comments.
FYI my last push is about reducing the roundtrips between C and Go.
I'm working on making the builds green again 😅

About the remaining discussion points:

• Listing sidekicks in config vs PHP-driven start:

The goal is for app developers to stay in control. Requiring both a PHP file and a Caddyfile entry for each sidekick is the wrong priority. With the app itself, it's three places to maintain for what should be an app-level concern. With PHP-driven start, adding a sidekick is just code: write the command, call get_vars('my-sidekick') from the worker bootstrap. No config change, no deployment diff. Third-party packages can ship their sidekicks; install the package, done. If someone needs to list running sidekicks, that's something the PHP app itself can expose (a debug command, a health endpoint). It doesn't need to be in the infrastructure config.

• Serialization:

String-only is a deliberate choice, not a limitation. Implicit serialization adds overhead, complexity, and a class of bugs: unserialize failures, class loading issues, exception confusion. APCu has had specific bugs from this in the past. For config publishing (which is what sidekicks do), strings are the right primitive. If structured data is needed, explicit encoding is better. Think SRP. For future task workers, serialization could make more sense (although personally I think this breaks SRP and APCu shouldn't have it) but should be designed separately with its own constraints.

• Unifying with task workers:

The non-HTTP worker foundation is already shared: same thread model, same crash recovery, same shutdown mechanism, same sidekick_entrypoint. The communication patterns are what differ: set_vars/get_vars for config publishing vs send_task/receive_task for work dispatch. These are complementary, not competing. A future task worker PR can reuse the same foundation and add its own communication primitives without touching the sidekick API. Forcing everything into one handle_task model would turn event-driven patterns (Redis pub/sub, file watching) into polling. Which is exactly what sidekicks exist to avoid.

• One more general thought on priorities:

I don't think task workers would solve new problems for the PHP ecosystem. We already have mature patterns for background work: Symfony Messenger, Laravel Queues, etc. These are actually better than in-process task workers because they offload work to separate processes, keeping the HTTP app focused on what it's for: serving users fast.

Sidekicks are the real frontier. The fundamental limitation of PHP's request/response model today is that apps can't receive pushed configuration updates. Every request has to pull its config: check Redis sentinel, pull vaults, evaluate feature flags. There's no way around it currently. Sidekicks change this: a background worker subscribes to changes and publishes them, so HTTP requests read pre-computed, always-fresh values with zero overhead. That's a capability PHP has never had before.

Of course, that's my take and I won't object tasks separately, but to me the focus here should be on getting sidekicks right, not on building a general task system that duplicates what existing queue libraries already do well.

Recent changes :

• I removed is_http_thread TLS variable entirely. Sidekick identity is now determined solely by sidekick_name (set via frankenphp_set_sidekick_name). Instead of two overlapping flags (is_worker_thread + is_http_thread), sidekick context is a single check: sidekick_name != NULL;

• I consequently removed is_http_thread guards from frankenphp_worker_request_startup and frankenphp_worker_request_shutdown: these are now unreachable from sidekicks because frankenphp_handle_request throws early if called from a sidekick context;

• Consequently also, I simplified frankenphp_update_local_thread_context to (bool is_worker): no more httpEnabled parameter;

• and I moved zend_unset_timeout() into frankenphp_set_sidekick_name().

Sidekick lifecycle is now separated from HTTP worker lifecycle:

• drainWorkerThreads now skips non-HTTP workers: sidekicks don't participate in the HTTP worker drain/restart cycle
• Sidekicks are no longer appended to the global workers slice; they're managed by their SidekickRegistry only
• Failed sidekick starts are now retryable: reserve/remove helpers on SidekickRegistry clean up the entry on failure instead of leaving a permanent broken entry

Readiness = set_vars, not script start:

• Sidekicks are now marked ready on the first successful set_vars call
• Boot failures stay classified as boot failures (backoff works correctly)
• Metrics now reflect actual sidekick readiness

I then refactored go_frankenphp_start_sidekick by extracting a Go startSidekick() function, which separates business logic from CGo export glue.

I optimized set_vars: vars are now stored as a persistent C HashTable instead of round-tripping through Go. get_vars builds the PHP array directly from the C data: one copy instead of the previous Go->C->PHP chain.

And there are new internal tests in sidekick_test.go covering retry semantics, drain skipping, readiness marking, and boot failure classification.

Recent changes :

• I removed is_http_thread TLS variable entirely. Sidekick identity is now determined solely by sidekick_name (set via frankenphp_set_sidekick_name). Instead of two overlapping flags (is_worker_thread + is_http_thread), sidekick context is a single check: sidekick_name != NULL;
• I consequently removed is_http_thread guards from frankenphp_worker_request_startup and frankenphp_worker_request_shutdown: these are now unreachable from sidekicks because frankenphp_handle_request throws early if called from a sidekick context;
• Consequently also, I simplified frankenphp_update_local_thread_context to (bool is_worker): no more httpEnabled parameter;
• and I moved zend_unset_timeout() into frankenphp_set_sidekick_name().

👍 👍 👍 👍

Sidekick lifecycle is now separated from HTTP worker lifecycle:

• drainWorkerThreads now skips non-HTTP workers: sidekicks don't participate in the HTTP worker drain/restart cycle

This needs to be changed, it's important that sidekicks restart on worker restart too, because the worker restart may be caused to clear opcache safely (PR open, not yet merged) or reload changed php files.

Readiness = set_vars, not script start:

• Sidekicks are now marked ready on the first successful set_vars call
• Boot failures stay classified as boot failures (backoff works correctly)
• Metrics now reflect actual sidekick readiness

That's what I wanted!

I then refactored go_frankenphp_start_sidekick by extracting a Go startSidekick() function, which separates business logic from CGo export glue.

Need to take a look later. I think most of my points have been addressed, but the last one I'd like to see is:

https://github.com/php/frankenphp/pull/2287/changes#r2952125541

Do we need an explicit entrypoint directive? Why can't the php logic define the entry point (as long as it's within the php_server's root?)?

it's important that sidekicks restart on worker restart too, because the worker restart may be caused to clear opcache safely (PR open, not yet merged) or reload changed php files.

Good call, PR updated! All green, except unrelated failures (could anyone maybe restart the failing job?)

Do we need an explicit entrypoint directive? Why can't the php logic define the entry point (as long as it's within the php_server's root?)?

Right, that's the last unanswered item. Here is my take:

Letting PHP define the entrypoint creates several problems:

• API leak: get_vars would need an $entrypoint argument. But sidekicks are keyed by name (at-most-once). Adding an entrypoint creates a tuple key, that opens for bugs where two call sites would have a different values.

• Viral config: the entrypoint path would have to be conveyed everywhere get_vars is called. That's application code carrying infrastructure details.

• Security surface: the entrypoint defines which scripts can run as long-running processes. Same reason worker { file ... } is in Caddyfile: it's an infrastructure boundary. Letting PHP choose arbitrary scripts to execute as persistent workers widens the attack surface.

• Consistency: HTTP workers have their entrypoint in Caddyfile. Sidekicks follow the same convention. One place for "what scripts run", another for "what they do".

The value proposition for the community is simple: add one line to your Caddyfile template, then everything else is pure PHP. The entrypoint never changes, it's the app's CLI runner. What sidekicks to start is the app's decision. For Symfony: sidekick_entrypoint /app/bin/console. Copy from docs, done. Same as worker { file /app/public/index.php }. Both are deploy-time decisions that belong in the infrastructure layer, not in application code.

I'm just wondering if it'd make sense to turn sidekick_entrypoint /app/bin/console into sidekick {file /app/bin/console }, or if this can be done later on if needed. WDYT?

Viral config: the entrypoint path would have to be conveyed everywhere get_vars is called. That's application code carrying infrastructure details.

Counter point: right now we have infrastructure carrying an application detail and our application code depends on infrastructure configuration. And while the single script is fine for Symfony, there are still plenty of projects that use more of a concerns-separated-by-folder structure.

Good call, PR updated! All green, except unrelated failures (could anyone maybe restart the failing job?)

Don't worry about it, this happens most CI runs. CI is green.

Same as worker { file /app/public/index.php }

This is kind of true, but only because there's no better alternative to define this in the application.

Personally I strongly believe in the model where the application should be the web server, but that's just not really a possibility in php yet. But where possible, I'd keep it in the application.

Fair points. I get that tension. The worker { file ... } precedent shows we've accepted this trade-off ;)

For projects with per-folder structures, sidekick_entrypoint can point to any dispatcher script, it doesn't force a Symfony-style bin/console. The routing-by-name happens inside the script, which is the app's domain. We have the public/index.php prececent, which is a pattern only WordPress doesn't follow (AFAIK).

I hear the "application as web server" vision. We're not there yet in PHP, but sidekicks are a step in that direction: the app decides what runs, infrastructure just provides the execution context. The entrypoint is the last piece that stays on the infra side. I think that's the right boundary for now.

The amphp project has building blocks for writing HTTP servers in pure PHP but the frankenphp model looks way more robust.

The sidekick { file ... } syntax I mentioned could leave room to evolve this later without breaking anything. But for now, a single flat directive keeps things simple.

Did we reach an agreement on this PR, with just a final review needed?

Fair points. I get that tension. The worker { file ... } precedent shows we've accepted this trade-off ;)

For projects with per-folder structures, sidekick_entrypoint can point to any dispatcher script, it doesn't force a Symfony-style bin/console. The routing-by-name happens inside the script, which is the app's domain. We have the public/index.php prececent, which is a pattern only WordPress doesn't follow (AFAIK).

With workers however, you can define multiple in multiple files or with multiple environments. With sidekicks, this wouldn't work. I actually "maintain" an application that has multiple workers defined (think admin/index.php, public/index.php and assets/generate.php)). With the current configuration I don't see how this would be possible with sidekicks and I would actually argue that it would make sense to keep sidekicks, which don't need the container, to be better written as simple scripts.

The entrypoint is the last piece that stays on the infra side. I think that's the right boundary for now.

I don't think I'm convinced here. but I'm not hard vetoing anything assuming the previous concern (see above) is addressed. I do believe that defining them within application code is the better idea, though - I don't think security is really an argument, given that the one script defined in infrastructure could do everything other files do, too.

The amphp project has building blocks for writing HTTP servers in pure PHP but the frankenphp model looks way more robust.

Swoole is quite advanced, but the general paradigm and libraries just aren't there yet. That's one of the reasons why I started contributing to FrankenPHP - I find it to bridge the PHP system in the right direction while making the compromises needed to make it happen. Found myself in the situation to choose between web servers in it's infancy and tried them all - it succeeded where Swoole, AmPhp/ReactPhp and Roadrunner fail.

Did we reach an agreement on this PR, with just a final review needed?

Mostly, I think. But I believe @AlliBalliBaba is most qualified to further handle feedback for this PR and the ultimate decision regarding such an extensive proposition lies with @dunglas. I haven't paid as much attention to the task worker background API as I should have, so I don't have the full picture on how to best design this approach. Also quite busy with work, so I only have chunks of time to throw at this while also juggling my other open source projects.

I'm very excited by this proposal, but I will not have the time to thoroughly review this PR until next week.

I'll likely raise some concerns with the proposed API/naming (it looks a bit specific to me, I'm pretty sure we can achieve something more generic with some small adjustments), but give me some time to dive deeper into the proposal.

I still dislike the points already mentioned in the design itself. The api allows for too many foot guns. Here's what needs to be addressed IMO.

• HTTP-threads shouldn't be able to start any number of global background threads that last and accumulate forever. The threads should either have a limited lifecycle or be managed by the go side (for example through caddy config).
• the background threads calling frankenphp_sidekick_set_vars should probably somehow be made a requirement. Even more ideal would be solving this with a more generic implementation of global variables or a global channel.
• background threads calling sleep seems to be a requirement in most cases, but it's hard to enforce.
• sleeping will also make potential worker restarts or shutdowns hang. Ideally the sleeping should be scheduled by the go side, similar to how http-threads go inactive when waiting for requests instead of frankenphp_sidekick_should_stop.
• You are saying that it's only possible to share strings between threads , but what you actually end up sharing is an array of strings. You should go all the way and allow sharing arrays directly (all the allocations happening currently are probably less efficient than serialization)
• I'd also rather have more generic names, maybe even a namespace.

Just as an example, this is how I could imagine what a configuration might look like, feels cleaner to have a central place where these background workers are defined. Each background worker gets 'ready' once they first listen to the ping coming from the go side.

frankenphp {
     sidekick_worker /path/to/redis_discoverer.php {
          ping 100ms # every 100ms
     }
     sidekick_worker /path/to/secret_vault.php {
          ping 5s # every 5s
     }
     sidekick_worker /path/to/cron.php {
          ping 60s "--domain=example.com" # every 60s with argv/argc or by just passing on the string
     }
}

# redis_discoverer.php

$discoverRedisEndpoint = function() {
    $redisEndpoint = ...;
    frankenphp_global_set('redisEndpoint', $redisEndpoint);
};

discoverRedisEndpoint();

while(frankenphp_handle_task($discoverRedisEndpoint)) { # 'ready' once we reach here
    # gets pinged periodically
}

With 'sidekick workers' a task is just scheduled periodically. But this way you're also potentially allowing an http worker to schedule a task directly.